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PREFACE 


This  effort  was  conducted  as  part  of  Project  2743,  Multiship  Research  and  Development 
(Multirad).  The  objective  of  Multirad  is  to  provide  qualitative  and  quantitative  data  that  will 
enhance  the  technology  base  and  for  acquiring  advanced  training  system  capabilities.  This 
technology  base  will  help  the  Air  Force  define  and  document  requirements  for  future  training 
systems,  identify  potential  training  benefits,  and  determine  simulation  complexity. 
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TRAINING  REQUIREMENTS  UTILITY  EVALUATION 


INTRODUCTION 

Today's  air  combat  pilot  faces  a  difficult  mission.  Situation  awareness  must  be  maintained  in 
spite  of  incomplete  and  often  inconsistent  data.  Enemy  aircraft  must  be  found  and  destroyed. 
Communications  and  electronic  jamming  must  be  overcome.  Enemy  weapons  must  be  defeated  using 
countermeasures  and  maneuvers.  In  addition,  the  pilot  must  successfully  execute  these  individual  tasks 
as  a  member  of  an  extended  team  of  warfighters. 

Training  for  this  mission  during  peacetime  is  expensive  and  difficult.  Most  combat  mission  training 
occurs  in  unit-level  continuation  training  programs.  These  programs  provide  the  apprenticeship 
environment  in  which  the  pilots  develop  their  skills  through  on-the-job  training  and  practice.  The  unit's 
continuation  training  program  gives  pilots  the  opportunity  to  learn  how  to  employ  their  weapon  systems 
effectively  and  to  practice  as  part  of  a  combat  team. 

While  the  cornerstone  of  a  unit's  continuation  training  program  is  in-flight  training,  several  factors 
combine  to  limit  in-flight  training  opportunities  for  many  combat  skills  (United  States  Air  Force  Scientific 
Advisory  Board,  1992).  These  factors  include  security  restrictions,  safety  of  flight  considerations, 
resources,  and  range  space.  In  addition,  resource  and  range  space  constraints  also  limit  the  opportunities 
for  collective  training  as  part  of  larger  force  units  (Defense  Science  Board,  1976;  1988).  These  training 
limitations  suggest  the  need  for  additional  training  in  many  key  mission  areas.  Operational  pilots,  training 
managers,  and  air  weapon  controllers  have  repeatedly  stated  their  desire  for  additional  training  in  many 
combat  mission  activities  (Gray,  Edwards,  &  Andrews,  1993;  Houck,  Thomas,  &  Bell,  1991).  Table  1 
shows  eight  of  the  combat  mission  activities  for  which  pilots  frequently  request  additional  training 
opportunities. 

Table  1 .  Mission  Activities  for  Which  Additional  Training  is  Desired 


Multi-bogey,  four  or  more 
Defense  against  all-aspect  threats 
Reaction  to  surface-to-air  missiles 
Dissimilar  air  combat  tactics 
Four-ship  tactics 
Reaction  to  air  interceptors 

Employment  of  electronic  countermeasures  and  electronic 
counter-countermeasures 
Chaff  and  flare  employment 


MCDONNELL  AIRCRAFT  SIMULATIONS 

Combat  engagement  simulations  allow  human  warfighters  and  their  simulated  weapon  systems  to 
engage  other  warfighters  on  a  virtual  battlefield.  These  simulations  are  becoming  increasingly  realistic 
because  of  the  continuing  advances  in  computer  and  communication  technologies.  Perhaps  the  best 
known  example  of  such  combat  engagement  simulations  is  the  simulator  networking  (SIMNET)  program. 
This  program  was  sponsored  by  the  Advanced  Research  Projects  Agency  (ARPA)  in  cooperation  with  the 
Army  and  showed  the  successful  use  of  combat  engagement  simulation  for  the  collective  training  of 
combat  units  (Alluisi,  1991). 

Based  on  the  potential  of  programs  like  SIMNET,  the  Armstrong  Laboratory  was  directed  in  the 
late  1980s  to  evaluate  multiship  simulation  for  air  combat  training.  The  laboratory  responded  to  this 
direction  with  a  combined  behavioral  and  engineering  program  conducted  with  the  cooperation  of  the  Air 
Combat  Command.  First,  we  identified  potential  weapon  systems  and  mission  activities  that  could  benefit 
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from  multiship  combat  simulation.  Next,  before  starting  a  long  and  potentially  expensive  development 
effort,  we  conducted  an  operational  training  utility  evaluation  using  existing  simulation  facilities.  The 
purpose  of  this  evaluation  phase  was  to  learn  if  ground-based  simulation  was  acceptable  for  combat 
mission  training.  To  reduce  costs  and  obtain  information  as  quickly  as  possible,  a  two-versus-many  air 
combat  simulation  was  used  at  McDonnell  Aircraft  Company  (MCAIR)  in  St.  Louis,  MO. 

MCAIR  System  Components 

The  simulation  system  used  at  MCAIR  was  designed  to  support  engineering  development.  Its 
design  and  equipment  represent  the  full-mission  simulator  facilities  developed  by  aircraft  manufacturers  in 
the  late  1980s.  Figure  1  shows  the  principal  components  of  this  system.  Each  F-15  cockpit  was  located 
in  a  40-ft  diameter  dome.  The  visual  world  was  created  by  a  combination  of  CompuScene  IV  image 
generators  and  video  target  projectors.  Enemy  surface-to-air  and  electronic  jammers  were  provided 
through  a  computer-based  threat  system.  Enemy  aircraft  consisted  of  both  piloted  and 
computer-controlled  adversaries.  The  manned  enemy  aircraft  were  flown  using  low  fidelity,  manned 
interactive  control  stations.  Human  participants  on  both  sides  were  supported  by  air  weapons  controllers. 


Figure  1.  McDonnell  Aircraft  Simulation  Components. 

Two  Gould  32/97  super  minicomputers  connected  by  an  Inter-Bus  Link  served  as  the  host 
computers  for  this  simulation.  The  simulators  and  other  peripheral  devices  were  connected  to  these 
computers  by  High  Speed  Devices  and  Ethernet.  A  more  detailed  description  of  this  simulation  is 
available  in  McDonnell,  Breeder,  and  Cutak  (1989). 
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Training  Utility  Evaluation 


Fourteen  weeks  of  air  combat  simulations  were  flown.  Each  week  consisted  of  four  days  of 
simulated  combat  missions.  An  average  of  twelve  pilots  and  six  air  weapons  controllers  took  part  in  each 
training  week.  The  participants  flew  as  a  formed  team  for  the  entire  week.  Each  formed  team  consisted 
of  a  flight  lead  and  wingman  from  the  same  fighter  squadron  and  an  air  weapons  controller.  The  team 
flew  at  least  one  simulator  sortie  each  day.  Each  simulator  sortie  lasted  about  one  hour  and  involved  a 
specific  air  combat  mission  (e.g.,  fighter  sweep,  point  defense,  force  escort). 

The  simulator  training  was  designed  using  a  building  block  approach  in  which  mission  difficulty 
was  manipulated  for  each  team  based  on  their  performance.  Mission  difficulty  was  controlled  by  varying 
threat  capabilities,  weather,  electronic  and  communication  jamming,  and  threat  tactics.  Each  team 
typically  flew  four  separate  combat  engagements  in  the  hour  sortie.  A  more  detailed  description  of  this 
evaluation  is  available  in  Houck,  Thomas,  and  Bell  (1991). 

Pilots  and  air  weapon  controllers  received  feedback  about  their  mission  performance  through  a 
variety  of  sources.  Within  each  simulator  all  systems  and  displays  responded  appropriately.  Engagement 
results  produced  real-time  outcomes  that  included  kill-removal  and  battle  damage.  The  visual  system 
allowed  the  pilot  to  see  missiles,  tracers,  other  aircraft,  and  explosions.  All  communications  between  the 
pilots  and  between  the  pilots  and  the  air  weapons  controller  were  recorded.  Each  F-15  pilot's  radar 
warning  receiver  and  radar  display  were  recorded  for  each  engagement.  This  information  was  combined 
with  a  plan-view  display  of  the  engagement  to  support  the  team's  post-mission  debrief.  In  addition,  an 
instructor  pilot  monitored  their  cockpit  instruments  and  voice  communications  from  the  test  director's 
station. 


The  responses  of  both  pilots  and  air  weapons  controllers  to  the  air  combat  simulations  were 
extremely  positive.  Table  2  shows  those  combat  activities  pilots  felt  were  better  trained  in  this  multiship 
simulation  than  in  their  unit  training  program.  These  include  the  combat  activities  that  are  difficult  to  train 
in-flight  because  of  resource  limitations,  security  restrictions,  and  safety  constraints.  The  pilots  also 
identified  several  combat  activities  for  which  their  unit  training  program  was  superior  to  this  multiship  air 
combat  simulation.  These  activities,  listed  in  Table  3,  require  high  visual  resolution.  Even  though  the 
MCAIR  domes  provided  state-of-the-art  target  and  laser  projection  systems,  they  still  could  not  provide  the 
pilots  with  enough  resolution  to  enable  them  to  see  the  visual  cues  they  rely  on  in  their  in-flight  training. 

Conclusions 

The  results  of  this  operational  training  utility  evaluation  show  the  potential  value  of  using 
simulators  to  help  train  pilots  and  air  weapons  controllers  in  air  combat  skills.  Both  Air  Force  pilots  and  air 
weapons  controllers  reported  that  multiship  air  combat  simulation  provided  better  training  for  many  combat 
activities  than  their  current  unit  training  program. 

DISTRIBUTED  INTERACTIVE  SIMULATION  IMPLEMENTATION 

Although  the  training  utility  evaluation  at  McDonnell  Aircraft  Company  showed  that  multiship 
simulation  could  complement  continuation  training  for  many  critical  combat  activities,  the  Air  Force  could 
not  afford  to  buy  and  maintain  a  similar  system.  Therefore,  more  affordable  alternatives  must  be 
developed  and  evaluated  if  multiship  simulations  are  to  become  an  integral  part  of  continuation  training. 

The  Multiship  Research  and  Development  (Multirad)  program  at  Armstrong  Laboratory,  Aircrew 
Training  Research  Division  (AL/HRA)  was  undertaken  to  create  a  system  that  would  provide  multiship  air 
combat  simulation  using  lower  cost,  distributed  simulation.  One  of  the  goals  of  the  Multirad  program  was 
to  develop  a  simulation  testbed  using  a  SIMNET-type  approach  that  would  support  training  effectiveness 
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Table  2.  F-15  Mission  Activities  for  Which  Multiship  Simulator  Training  was 

Rated  Significantly  Higher  than  Unit  Training 


Multi-bogey,  four  or  more 
Reaction  to  surface-to-air  missiles 
Dissimilar  air  combat  tactics 

Employment  of  electronic  countermeasures  and  electronic 
counter-countermeasures 
Defense  against  all-aspect  threats 
Escort  tactics 
All-weather  employment 
Communications  jamming 
Low  altitude  tactics 
Threat  warning  assessment 


Table  3.  F-15  Mission  Activities  for  Which  Muitiship  Simulator  Training  was 

Rated  Significantly  Lower  than  Unit  Training 


Visual  lookout 
Tactical  formation 
Visual  identification 
Mutual  support 


research.  Once  the  simulators  were  integrated  into  the  Multirad  distributed  simulation  network,  a  series  of 
training  exercises  known  as  the  Training  Requirements  Utility  Evaluation  (TRUE)  was  conducted.  The 
purpose  of  the  TRUE,  which  replicated  many  elements  of  the  MCAIR  air  combat  simulations,  was  to  find 
out  if  distributed  simulation  provided  an  acceptable  simulation  of  two-versus-many  air  combat. 

Multirad  System  Components 

The  Multirad  system  consists  of  several  independent  systems  connected  via  network  interface 
units  (NIUs)  which  convert  each  device’s  unique  codes  into  a  common  communication  protocol.  The 
components  used  in  the  TRUE  were:  two  F-15  cockpits,  computer  image  generation  and  displays  for  the 
F-15s,  two  opposing  forces  cockpits,  an  air  weapons  controller  station,  a  computer-generated  threat 
system,  an  exercise  control  and  video  recording  station,  and  a  separate  video  debriefing  station  (see  Fig. 
2). 


Figure  2.  Multiship  Research  and  Development  (Multirad)  System  Network. 
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F-15  Cockpits 


The  two  F-15C  cockpits  used  were  McDonnell  Douglas  Reconfigurable  Cockpits  (MDRCs).  The 
MDRC  incorporates  high  fidelity  stick  and  throttle  grips  with  a  color  CRT/touchscreen  depiction  of  the  front 
panel  (see  Fig.  3).  The  MDRC  uses  commercial,  off-the-shelf,  VME-based,  microprocessors  to  perform 
all  internal  functions.  Inside  its  single  VME  chassis,  the  MDRC  has  four  Motorola  68030  single  board 
computers,  one  Motorola  quad-processor  (88100)  computer  board,  two  computer  image  generator  board 
sets,  a  sound  board,  and  several  digital-to-analog  and  analog-to-digital  converters.  The  two  image 
generators  are  used  to  provide  the  cockpit  instruments  and  the  head-up  display  (HUD).  The  sound  board 
provides  weapons  cues  and  aircraft  audio  such  as  engine  sounds  and  g-limit  warnings. 

The  MDRC  has  a  high  fidelity  F-15  software  suite  that  is  derived  directly  from  the  McDonnell 
Douglas  engineering  simulators  in  St.  Louis.  The  software  includes  an  F-15  aerodynamics  package,  a  full 
assortment  of  air-to-air  weapons,  a  complete  radar  package,  a  radar  warning  receiver  (RWR),  a  HUD, 
electronic  countermeasures  (ECM),  and  electronic  counter-countermeasures  capabilities  (ECCM).  The 
MDRC  provides  high  fidelity  simulation  only  for  air  combat  functions;  there  are  no  rudder  pedals  or 
provisions  for  takeoff,  landing,  refueling,  or  emergency  procedures. 

F-15  Visuals 

Computer  Image  Generation  -  Imagery  was  provided  by  the  General  Electric  Advanced  Visual 
Technology  System  (AVTS)  which  was  the  engineering  prototype  for  the  CompuScene  4.  AVTS  provides 
8,000  faces  distributed  among  ten  channels  at  60  Hz. 

Displays  -  One  MDRC  was  installed  in  the  McDonnell  Douglas  full  field-of-view  dome.  This 
system  consists  of  a  24-ft  diameter  dome  with  360  deg  horizontal  by  190  deg  vertical  coverage  (Reno, 
1989).  The  display  system  incorporates  six  background  projectors  with  a  resolution  of  4.3  arc-min/pixel 
and  a  40  deg,  head-tracked  area  of  interest  (AOI)  (see  Fig.  4). 


Figure  3.  McDonnell  Douglas  Reconfigurable  Cockpit  (MDRC)  Installed  in  the  Display  for  Advanced 

Research  and  Training  (DART). 


Resolution  in  the  AOI  was  2.4  arc-min/pixel.  Only  the  AOI  and  the  three  forward  channels  were 
used  during  the  TRUE  resulting  in  an  210  deg  horizontal  by  100  deg  background  field  of  view.  The  AOI 
could  be  slewed  throughout  the  dome.  The  other  MDRC  was  installed  in  the  Armstrong  Laboratory 
Display  for  Advanced  Research  and  Training  (DART).  The  DART  is  a  dome-like  display  system 
consisting  of  eight  segments  of  a  dodecahedron  which  surround  the  cockpit  (Thomas,  Reining,  &  Kelly, 
1991).  Each  segment  is  a  rear-projection  screen  approximately  1m  from  the  pilot's  head  (Fig.  3).  During 
the  TRUE,  imagery  was  projected  onto  only  six  of  the  screens  at  a  time  as  controlled  by  a  head  tracker. 
The  DART'S  field  of  view  during  the  TRUE  was  300  deg  horizontal  by  200  deg  vertical  with  resolution  of 
4.75  arc-min/pixel.  Unlike  the  dome  pilot,  the  DART  pilot  could  not  see  to  the  rear  of  the  aircraft. 


Figure  4.  Interior  of  the  Full  Field-of-View  Dome  (Cockpit  has  been  removed). 

Opposing  Forces  Cockpits 

Two  Armstrong  Laboratory  Combat  Engagement  Trainers  (CETs)  were  integrated  into  the 
network  as  opposing  airborne  interceptors  (Fig.  5).  CETs  are  limited-function  F-16  trainers  equipped  with 
air-to-air  radar,  AIM-9  missiles,  and  radar  warning  receivers  (Boyle  &  Edwards,  1992).  For  the  TRUE,  the 
F-16  aerodynamics  simulation  was  replaced  with  aerodynamic  and  engine  characteristics  of  an  Su-27 
(Flanker)  interceptor. 

Air  Weapons  Controller  Station 

The  Simulated  Command  and  Control  Environment  Networked  Training  System  (SCCENTS)  was 
developed  by  the  Logistics  Research  Division  of  Armstrong  Laboratory  to  provide  the  Air  Force  with  a  low- 
cost  command  and  control  workstation  for  research,  development,  and  training  (Fig.  6).  The  system  was 
designed  to  be  integrated  to  networks  such  as  SIMNET  or  Distributed  Interactive  Simulation  (DIS).  The 
SCCENTS  has  two  display  modes  -  an  Airborne  Warning  and  Control  System  (AWACS)  or  the  407L 
Ground  Control  Intercept  (GCI)  display.  The  AWACS  display  was  used  for  TRUE  research  because  it 
provided  more  physical  information  about  the  battle  to  the  controllers.  The  SCCENTS  also  has  a 
geographic  database  developed  from  Defense  Mapping  Agency  data. 


6 


Figure  5.  Limited  Function  Combat  Engagement  Trainer  (GET) 


Computer-Generated  Threats 

The  F-15  and  opposing  forces  cockpits,  visual  image  generators,  displays,  and  the  controller 
station  were  existing  systems  which  were  adapted  for  network  operation.  The  automated  threat 
engagement  system  (ATES)  was  developed  especially  for  the  Multirad  effort  (Rogers,  1992).  The  ATES 
simulates  ground  and  air  threats  plus  friendly  aircraft.  Ground  threats  include;  headquarters  functions  with 
early  warning  radar,  directed  and  autonomous  surface-to-air  missile  (SAM)  batteries  (SA-4,  SA-6,  and  SA- 
8)  with  their  radars,  and  ZSU-23  anti-aircraft  artillery  (AAA).  ATES  air  threats  used  in  the 


Figure  6.  Simulated  Command  and  Control  Environment  Networked  Training  System  (SCCENTS)  Used 

as  the  Air  Weapons  Controller  Station. 
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TRUE  were  Su-27  interceptors  armed  with  radar  and  infrared  guided  missiles  and  MiG-27  (Flogger)  attack 
fighters  equipped  with  radar  jammers.  ATES  also  supplied  four  F-16  strike  fighters  used  during  offensive 
counterair  missions.  While  the  ATES  hardware  was  specifically  developed  for  Multirad,  the  threat  models 
and  integrating  software  are,  "a  blend  of  several  programs  from  various  government  agencies  and  a 
commercial  vendor,"  (Rogers,  1992,  p.  303). 

Exercise  Control 

Multiship  simulation  exercises  were  directed  from  a  central  console  which  contained  systems  to 
set  up,  initiate,  observe,  videotape,  and  terminate  sorties.  Initial  conditions  and  actions  of  automated 
opposing  forces  were  preprogrammed  in  the  ATES.  Initial  conditions  for  all  manned  players  are 
programmed  in  the  exercise  control  station.  Monitors  at  the  control  station  displayed;  (a)  an  overhead 
view  of  the  engagement,  (b)  each  F-15's  front  panel  including  radar,  radar  warning,  and  armament  control 
displays,  and  (c)  one  channel  of  out-the-window  video  (the  AOI  for  the  dome  and  the  forward  channel  for 
the  DART)  (Fig.  7).  The  test  director  had  intercom  communication  with  each  player  station.  Incorporated 
into  the  exercise  control  station  was  a  computer-controlled,  videotaping  system  which  recorded  the  two  F- 
15  front  panels  and  the  overhead  view  plus  all  radio  communication.  Computer  control  allowed 
synchronized  start,  stop,  and  playback  of  the  three  videotapes. 


Figure  7.  Multirad  Exercise  Command  and  Control  Station. 


Debriefing  station 

After  each  simulator  session,  the  two  F-15  pilots  and  the  air  weapons  controller  would  take  the 
videotapes  to  an  independent  debriefing  room  which  contained  three  computer-controlled  tape  players 
and  monitors  (Fig.  8).  After  brief  instruction,  participants  were  able  to  review  their  engagements  and  could 
stop,  rewind,  and  replay  segments  of  particular  interest.  Since  the  debriefing  system  was  independent  of 
the  simulators,  a  second  team  could  fly  while  the  first  was  debriefing. 
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Figure  8.  Multirad  Video  Debriefing  System. 

System  Development  and  Integration 


Network  Interface  Units 

The  Network  Interface  Unit  provides  a  method  of  communicating  between  the  host  simulator  and 
the  Multirad  network  (SIMNET).  The  NIU  and  host  communicate  with  one  another  according  to  a 
predefined  language  which  is  described  in  the  system's  interface  control  document  (ICD).  The  NIU  uses 
ethernet  as  the  physical  medium  to  communicate  both  with  the  host  computer  and  SIMNET  although  other 
mediums  such  as  Fiber  Distributed  Data  Interface  (FDDI)  and  reflective  memory  are  available  for  use.  The 
NIU's  primary  functions  include  coordinate  system  conversion,  remote  vehicle  approximation  (RVA),  data 
filtering,  and  conversion  of  units  of  measure.  RVA  algorithms  were  used  to  reduce  the  amount  of  data  on 
the  Multirad  network.  The  algorithms  use  linear  interpolation  to  determine  a  vehicle's  new  position.  All 
moving  entities  on  the  network  had  RVA  models  of  10  m  long,  20  m  wide,  1  m  high,  and  3  deg  rotational. 
If  a  model’s  delta  between  the  actual  position  and  RVA  position  exceeded  10%  for  any  of  these 
dimensions,  an  appearance  packet  was  sent  to  all  other  vehicles  on  the  network. 

Armstrong  Laboratory  developed  one  NIU  for  each  of  its  networked  simulation  systems.  The 
NIU's  hardware  consists  of  two  Motorola  68030  single  board  computers  with  transition  modules,  an 
enhanced  ethernet  interface  board  from  CMC  Corporation,  a  SIMVAD  digital  voice  processing  board,  a 
SIMVAD  voice  communications  adapter  with  a  headset,  a  twelve-slot  VME  chassis,  and  a  removable  disk 
drive  assembly.  One  computer  board  processed  simulation  data  between  the  SIMNET  network  and  the 
simulation  host  while  the  other  board  processed  digital  voice  data  between  the  host  and  the  network.  The 
CMC  ethernet  processor  board  was  chosen  for  communications  between  the  NIU  and  SIMNET  because  it 
was  faster  than  the  MVME712  transition  module  interface.  The  CMC  boards  also  contained  firmware  to 
monitor  ethernet  statistics  such  as  collisions  and  bad  packets. 

Communications  Protocol  and  Extensions 

SIMNET  was  developed  by  the  Army  and  ARPA  for  tanks  and  slow  moving  air  vehicles. 
Armstrong  Laboratory  implemented  a  subset  of  the  SIMNET  Protocol  Data  Units  (PDUs)  specific  to  air 
combat  operations.  Initially,  the  activate  request  and  response,  deactivate  request  and  response,  vehicle 


9 


appearance,  fire,  and  impact  PDUs  were  implemented  and  tested.  After  all  systems  were  able  to 
communicate  with  each  other  and  observe  each  other  in  the  synthetic  battlefield,  the  protocols  were 
extended  to  include  freeze,  radar,  and  emitter  PDUs.  The  MDRCs  provided  the  most  fidelity  and  were 
chosen  to  provide  the  testbed  for  implementing  the  protocol  extensions.  The  MDRCs  were  first  tested  one 
on  one,  then  integrated  to  the  other  devices  on  the  network. 

All  SIMNET  PDUs  were  modified  to  add  a  timestamp  field.  Additionally,  fifteen  extra  result  types 
were  added  to  the  impact  PDU  to  enhance  scoring  capabilities.  Originally,  the  SIMNET  protocol  only  had 
four  results  for  scoring;  miss,  ground  impact,  vehicle  impact,  and  proximity  impact.  A  freeze  PDU  was 
also  implemented  to  allow  the  exercise  control  station  to  stop  (freeze),  and  continue  any  mission. 

The  radar  and  emitter  PDUs  were  designed  to  pass  all  the  information  needed  about  a  specific 
radar  or  emitter  over  the  network  to  other  vehicles.  The  radar  PDU  includes  radar  system,  radar  mode, 
radar  ID,  sweep,  power,  and  a  list  of  illuminated  targets.  The  emitter  PDU  includes  the  number  of 
emitters,  emitter  class,  mode,  power,  frequency,  and  sweep.  Air-to-air  Tactical  Aid  to  Navigation 
(TACAN),  Identification  Friend  or  Foe  (IFF),  jammer,  and  radio  emitters  were  specifically  implemented  on 
the  network.  One  issue  that  had  to  be  addressed  was  the  classified  nature  of  the  radars  and  emitters  of 
the  MDRCs.  Rather  than  send  classified  data  over  the  network,  all  players  had  classified  information 
about  all  other  players  on  the  net.  In  this  way,  PDUs  were  implemented  such  that  individual  packets  were 
not  classified  but  aggregates  of  packets  may  be  classified. 

Each  system’s  level  of  computing  ability  was  different.  Initially,  the  MDRCs  originally  could  only 
handle  8  threat  vehicles,  8  missiles,  and  8  ECM  bodies.  Upgrading  the  MDRCs  to  Motorola  88100 
processors  increased  capability  to  support  15  threats,  8  network  missiles,  8  internal  missiles,  8  network 
ECM  bodies,  8  internal  ECM  bodies,  and  8  SAM/AAA  sites.  The  CETs  on  the  other  hand  could  only 
handle  6  network  vehicles.  A  priority  routine  was  used  to  monitor  the  closest  threats.  Similar  limitations 
existed  in  the  visual  systems  where  priority  schemes  were  also  implemented. 

While  integrating  the  MDRC  to  the  network,  a  single  vehicle  update  message  was  designed  to 
transfer  simulation  information  about  all  entities  on  the  network.  This  single  vehicle  update  message 
contained  position,  velocity,  and  state  information  including  radars  and  emitters.  However,  ECM  and 
missile  entities  do  not  need  to  pass  information  such  as  radars,  emitters,  and  throttle  position.  A  second, 
streamlined,  vehicle  appearance  message  was  created  for  ECM  and  missiles  to  reduce  the  amount  of 
traffic  between  the  host  and  NIU. 

Problems  Encountered 

Many  unexpected  problems  arose  during  the  integration  of  the  systems  for  the  TRUE.  For 
example,  the  CETs  and  ATES  were  accepting  any  impact  message  containing  their  vehicle  ID  as  a  kill 
without  looking  at  the  result.  Under  certain  circumstances,  the  MDRCs  would  kill  vehicles  when  the  gun 
trigger  was  depressed-even  though  the  vehicle  was  80  miles  away.  Ghost  vehicles  were  also  created  on 
the  network  due  to  improper  memory  management.  Still  another  memory  management  problem  allowed 
vehicles  to  attain  attributes  of  vehicles  that  were  previously  in  that  portion  of  memory.  A  CET  might  fly  a 
jamming  mission  in  one  sortie;  in  the  next  sortie,  whichever  vehicle  occupied  the  same  memory  space  had 
the  jammer  flag  set. 

During  development  for  TRUE,  networked  vehicles  jittered  in  the  visual  systems.  Several  reasons 
for  jitter  were  identified.  One  portion  of  the  jitter  occurred  when  a  vehicle  exceeded  the  RVA  dimensions. 
Jitter  was  also  caused  by  the  different  frequency  rates  of  the  devices  on  an  asynchronous  network.  Still 
another  cause  for  jitter  was  simulator  or  NIU  overloading.  When  a  device  could  not  send  and  receive 
packets  at  its  predefined  update  rate,  it  would  take  large  jumps  in  the  visual  systems.  Finally,  coordinate 
conversions  and  precision  were  found  to  be  contributors  to  jitter.  In  the  early  stages  of  TRUE 
development,  some  vehicles  were  making  large  jumps  in  the  visuals.  The  coordinate  transformation 
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algorithms  in  the  NIUs  were  found  to  be  incorrect.  All  systems  were  analyzed  and  modified  to  reduce  the 
possibility  of  overloading.  The  scenarios  were  also  analyzed  to  ensure  devices  did  not  overload  due  to 
network  traffic.  Additionally,  the  coordinate  transformations  were  corrected.  The  resulting  jitter  was 
deemed  acceptable  to  the  program  because  it  was  minor.  A  smoothing  algorithm  could  be  used  to  further 
reduce  the  jitter  but  was  not  implemented  due  to  system  loading. 

Test  and  Evaluation 

TRUE  was  conducted  as  a  series  of  air  combat  training  exercises  similar  to  the  exercises 
conducted  at  McDonnell  Aircraft  Company.  Three  or  four  teams  of  lead  and  wing  F-15  pilots  plus  an  air 
weapons  controller  participated  in  four,  week-long  exercises.  During  each  of  these  weeks,  teams  flew 
seven,  one-hour  simulator  sessions.  During  each  session,  teams  flew  three  or  four  setups  of  either  a 
defensive  (DCA)  or  offensive  (OCA)  counter-air  mission.  On  DCA  missions,  the  team  defended  their 
home  airbase  against  an  attack  from  two  MiG-27  (Flogger)  fighter-bombers  escorted  by  four  Su-27 
(Flanker )  fighters  (two  manned  and  two  computer-generated).  On  OCA  missions,  the  F-15s  escorted  a 
flight  of  four  computer-generated  F-16s  attacking  the  air  base  which  was  defended  by  six  Su-27  fighters, 
four  computer-generated  and  two  manned.  Each  DCA  or  OCA  setup  was  initiated  with  the  aircraft  at 
20,000  ft  separated  by  80  nmi.  Computer-generated  enemy  force  tactics  were  preprogrammed  and  the 
manned  players  followed  a  script  during  the  early  part  of  each  engagement.  There  were  six  variations  of 
enemy  tactics  for  the  OCA  mission  and  seven  for  the  DCA  mission.  In  addition,  the  manned  players  were 
free  to  deviate  from  the  script  as  circumstances  demanded  and  the  computer-generated  aircraft  were 
programmed  to  defend  themselves  when  attacked.  Figures  9  and  10  illustrate  two  variations  of  the  DCA 
and  OCA  missions.  F-1 5  pilots  and  controllers  were  not  shown  the  enemy  force  plans  nor  were  they 
informed  about  which  variation  would  be  seen  on  any  setup.  Variations  were  selected  at  random  with  the 
provision  that  no  variation  was  repeated  within  a  given  simulator  session. 


DCA1 : 6-Ship  Champagne 

Flanker  Flogger 
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Figure  9.  Two  Variations  of  the  Defensive  Counterair  (DCA)  Missions  Used  in  the  Training  Requirements 

Utility  Evaluation  (TRUE). 
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Figure  1 0.  Two  Variations  of  the  Offensive  Counterair  (OCA)  Missions  Used  in  the  T raining 

Requirements  Utility  Evaluation  (TRUE). 

At  the  beginning  of  each  week,  pilots  and  controllers  were  briefed  about  the  TRUE  objectives  and 
procedures.  Participants  then  filled  out  a  questionnaire  listing  30  air  combat  mission  tasks  and  skills  and 
were  asked  to  rate  the  desirability  of  receiving  additional  training  on  each.  After  a  familiarization  flight, 
teams  flew  three  DCA  and  three  OCA  simulator  sessions  with  three  or  four  setups  per  session.  Before 
each  session,  teams  planned  their  missions  including  call  sign  assignment,  lookout  responsibilities,  and 
plans  for  attack,  mutual  support,  and  re-attack.  Each  engagement  was  videotaped  from  the  exercise 
control  console  using  three,  computer-controlled  recorders.  Each  F-15  cockpit’s  front  panel  including  the 
radar,  radar  warning,  and  armament  control  displays  was  recorded  along  with  the  overhead  view  of  the 
engagement  from  the  control  console  plus  all  audio  communications  and  warnings.  At  the  end  of  the 
simulator  session,  teams  took  the  tapes  to  an  independent  debriefing  room  which  contained  facilities  for 
synchronized  playback  of  the  tapes.  After  debrief,  each  team's  lead  pilot  completed  a  questionnaire 
describing  any  difficulties  they  experienced  during  the  missions  and  lessons  learned  for  the  next  simulator 
session.  Participant  comments  and  critiques  were  also  solicited  during  daily  meetings  with  all  pilots  and 
controllers.  During  some  TRUE  weeks,  pilots  flew  one-vs-one  engagements  between  the  two  F-15 
cockpits.  After  all  simulator  sessions  had  been  completed,  participants  filled  out  a  final  questionnaire.  On 
this  questionnaire,  participants  rated  the  quality  of  training  received  in  their  current  unit  training  program 
and  from  Multirad  for  each  of  30  tasks  and  skills. 

TRUE  Participants 

Twenty-three,  USAF,  F-15  pilots  and  13  air  weapons  controllers  participated  in  TRUE  exercises. 
Pilot  experience  levels  ranged  from  300  to  2,500  total  flying  hours  with  a  median  of  1,400  total  hours  and 
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675  F-15  hours.  The  distributions  of  controller  experience  and  flight  hours  of  TRUE  pilots  by  unit 
qualification  are  shown  on  Figures  11  and  12. 
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Figure  1 1 .  Distribution  of  TRUE  Air  Weapons  Controller  Experience. 
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Figure  12.  Distribution  of  TRUE  Pilot  Experience  by  Unit  Qualification. 

Results 

During  the  four  weeks  of  the  TRUE,  78  h  of  multiship  simulator  exercises  were  scheduled  and  72 
were  completed;  6  h  (8%)  were  lost  to  major  systems  failures.  Within  the  72  h,  267  multiship  setups  were 
conducted.  Of  these,  204  setups  were  completed  successfully  while  63  (24%)  required  a  restart  due  to 
minor  system  failure.  The  proportion  of  setups  which  experienced  minor  failures  dropped  from  30%  during 
the  first  week  to  21%  during  the  other  three  weeks. 

Network  Analysis 

Network  traffic  was  captured  for  the  final  week  of  the  TRUE  to  determine  network  loading  and 
characteristics.  Table  4  summarizes  the  data  captured  for  the  three  engagement  types  used  in  the  TRUE: 
basic  fighter  maneuvers  (BFM),  defensive  counter-air  (DCA),  and  offensive  counter-air  (OCA).  The  BFM 
engagements  were  one-on-one  fights  between  the  two  MDRCs.  The  average  utilization  for  all 
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engagements  was  less  than  one  percent  of  network  capacity  while  the  maximum  utilization  was  never 
more  than  3.4%.  Additionally,  the  network  exceeded  90%  of  the  maximum  values  only  two  to  three  times 
per  engagement.  Network  analysis  showed  no  collisions,  no  cyclic  redundancy  check  errors,  and  no 
alignment  errors.  This  was  probably  due  to  the  fact  the  network  was  not  heavily  loaded. 

Table  5.  Average  Network  Packet  Utilization 
Table  4.  Multirad  Network  Utilization  (Kbits/s).  (Pkts/s). 
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204 

294 

87 

Appearance 

30 

38 

47 

Avg 

50 

73 

87 

Radar 

1 

4 

4 

Emitter 

0 

16 

16 

( 

Voice 

5 

16 

16 

The  largest  number  of  PDUs  on  the  network  was  the  vehicle  appearance  PDU  followed  by  the 
voice,  radar,  and  emitter  PDUs  (see  Table  5).  The  deactivate,  fire,  and  impact  PDUs  are  all  event-type 
PDUs  and  each  took  less  than  one  packet  per  second  during  the  TRUE.  The  network  loading  is 
consistent  for  the  three  different  scenarios.  The  number  of  entities  on  the  network  is  dependent  on  pilot 
input.  The  more  ECM  and  missiles  are  deployed,  the  more  entities  are  placed  on  the  network.  The 
average  number  of  entities  for  each  type  engagement  was  24  entities  for  each  BFM,  38  entities  for  each 
DCA,  and  39  entities  for  each  OCA.  RVA  algorithms  reduced  network  traffic  by  65%  to  85%  over  the 
course  of  the  TRUE.  Because  tactics  and  maneuvers  are  different  for  each  engagement,  the 
effectiveness  of  RVA  on  network  loading  was  also  different  for  each  engagement. 

Although  network  performance  was  near  perfect,  a  hardware  discrepancy  in  some  of  the  NIU's 
ethernet  boards  was  discovered  early  in  the  program  that  affected  network  reliability.  The  discrepancy 
was  a  result  of  the  fact  that  the  only  contacts  between  the  circuit  board  and  the  connector  were  solder 
joints  on  the  pins.  After  repeated  use  of  the  connectors,  the  solder  Joints  would  fail,  resulting  in  loss  of 
data.  This  loss  of  data  caused  unpredictable  operations  on  the  network  and  large  amounts  of  Jitter.  Once 
the  connectors  were  repaired  by  the  manufacturer,  the  problem  was  resolved. 

Component  Utility 

F-15  Cockpits  -  The  MDRC  cockpits  used  by  the  TRUE  pilots  were  rated  as  wholly  acceptable  for 
air  combat  training.  The  glass  cockpit  and  touch  panel  were  downrated  only  in  that  the  displays  were  not 
positioned  exactly  as  in  the  aircraft,  and  pilots  had  to  scan  for  a  moment  to  find  what  they  were  looking  for. 
The  lack  of  rudder  pedals  was  cited  as  a  problem  only  in  close  combat  which  was  not  a  Multirad  objective. 
A  major  difficulty,  however,  was  that  some  of  the  avionics  software  in  the  simulator  was  not  current  with 
the  aircraft.  Pilots  complained  vigorously  that  the  older  software  prevented  them  from  using  their  weapons 
systems  as  they  would  in  the  aircraft.  This  lack  of  currency  affected  the  pilots'  tactics  and  greatly  reduced 
the  value  of  Multirad  training. 

Visual  Display  Systems  -  The  offensive  and  defensive  counterair  missions  were  designed  to 
emphasize  beyond-visual-range,  air-to-air  combat  tasks:  e.  g.,  radar  searching,  sorting,  targeting, 
weapons  employment,  communication,  tactics,  and  mutual  support.  However,  pilots  uniformly  reported 
that  they  experienced  significant  difficulties  in  completing  their  missions  due  to  problems  with  the  visual 
displays.  Both  the  dome  and  the  DART  received  high  praise  for  depicting  the  terrain  and  horizon.  This 
information  was  used  for  Judging  attitude  or  for  low  altitude  flight  when  necessary.  Difficulties  were 
primarily  experienced  in  seeing  other  aircraft.  An  instaneous  field  of  regard  of  less  than  180  deg 
prevented  pilots  from  maintaining  tactical  formation  as  they  normally  do  in  the  aircraft,  i.  e.,  at  3  or  9 
o'clock.  The  AOI  in  the  dome  received  particular  criticism.  A  pilot  in  the  dome  could  not  see  his  wingman 
in  tactical  formation  using  either  peripheral  vision  or  a  quick  saccade  left  or  right.  The  pilot  in  the  dome 
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had  to  turn  his  head  90  deg  and  spend  a  second  or  two  looking  for  his  wingman.  DART  pilots 
experienced  less  difficulty  in  maintaining  tactical  formation  within  1  nmi.  Problems  with  field  of  view  also 
prevented  pilots  from  providing  mutual  support  while  engaged  with  enemy  aircraft. 

Pilots  described  their  ability  to  resolve  an  air  target  as  acuity  which  is  a  function  of  display 
brightness,  contrast,  and  resolution,  plus  the  level  of  detail  in  the  computer-generated  aircraft  model. 
Level  of  detail  varied  with  range  from  the  eyepoint.  The  major  problem  caused  by  lack  of  acuity  was 
inability  to  determine  the  aspect  of  other  aircraft.  Pilots  could  not  determine  in  which  direction  other 
aircraft  were  heading  without  watching  their  target's  flight  path  for  several  seconds.  Pilots  also  reported 
that  they  could  not  maintain  tactical  formation  beyond  0.5-1  nmi  without  using  many  radio  calls  due  to 
inability  to  judge  aspect.  In  addition,  pilots  could  not  visually  determine  distance  from  other  aircraft  at  low 
levels  of  detail.  Acuity  problems  also  plagued  pilots  at  the  merge  where  the  F-15  passes  an  enemy 
aircraft  at  high  speed.  When  a  pilot  could  visually  locate  the  threat  aircraft  or  his  wingman  he  could  not  tell 
which  way  the  other  aircraft  were  going  quickly  enough  to  take  a  tactical  advantage.  Overall,  both  DART 
and  dome  pilots  reported  that  visual  flying  was  difficult  and  that  they  learned  to  use  nighttime  tactics  and 
detached  mutual  support. 

To  increase  the  range  at  which  aircraft  could  be  detected  and  identified,  a  simulator-unique  effect 
(i.  e.,  a  simism)  was  introduced.  At  maximum  visual  detection  range,  an  air  target  was  represented  as  a 
white  point  light.  At  maximum  identification  range,  enemy  aircraft  were  represented  as  red  point  lights 
while  friendlies  were  represented  as  blue,  flashing  lights.  This  simism  greatly  increased  pilot  acceptance; 
however,  pilots  continued  to  complain  that  they  could  not  determine  another  aircraft's  range  or  aspect  until 
it  was  within  0.5-1  nmi. 

At  the  end  of  each  week,  pilots  were  asked  to  rate  the  value  of  the  training  received  using  the 
Multirad  system  for  each  of  30  flight  tasks.  Multirad  training  was  rated  highest  for  nonvisual  tasks, 
notably,  operation  against  multiple  enemy  aircraft  and  practice  working  with  an  air  weapons  controller. 
Among  the  visual  tasks.  Multirad  training  was  rated  highest  for  dissimilar  air  combat  training  and  defense 
against  SAMs.  Mean  ratings  for  the  DART  and  Dome  are  shown  on  Figure  1 3.  The  tasks  are  coded: 


DACT 
SAM  DEFENSE 
TWO  SHIP  TAG 
VISUAL  LOW 
VISUAL  LOOK 
MUTUAL  SUPT 
LOW  ALT  TAG 
VISUAL  ID 
TAGTIGAL  FRM 


Dissimilar  air  combat  training 

Defense  against  surface-to-air  missiles 

Two-ship  tactics 

Visual,  low  altitude  flight 

Visual  lookout 

Mutual  support 

Low  altiude  tactics 

Visual  identification  of  target  aircraft 

Tactical  formation. 


Differences  in  ratings  between  the  two  displays  are  not  significantly  different. 

Twelve  pilots  participated  in  DART  vs  dome  BFM  exercises.  Aircraft  were  initialized  facing  each 
other  10  nmi  apart.  Pilots  were  instructed  to  fly  towards  each  other  and  turn  to  engage  as  they  passed. 
After  three  setups,  pilots  traded  cockpits  and  flew  three  more.  Overall,  36  out  of  55  engagements  (65%) 
were  won  by  the  pilot  flying  in  the  DART. 
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DACT 
SAM  DEFENSE 
TWO  SHIP  TAG 
VISUAL  LOW 
VISUAL  LOOK 
MUTUAL  SUPT 
LOW  ALT  TAG 


bgSSSggggS888688ag^^ 


VISUAL  ID 


TAGTIGALFRM 


^  DART 
□  Dome 


4 


Unacceptable  Rating  of  Multirad  Training  Good 


Figure  13.  Comparison  of  Mean  Ratings  for  Multirad  Training  for  the  DART  and  Dome  Visual  Displays  on 

Visually  Demanding  Tasks. 

Opposing  Forces  -  Houck  et  al.  (1991)  found  in  the  Advanced  Air  Combat  Training  program  that 
the  most  significant  training  benefits  came  from  the  opportunity  to  engage  multiple  bogeys.  However,  in 
the  first  week  of  the  TRUE,  participants  found  so  many  flaws  in  the  representation  of  opposing  aircraft  that 
they  rated  the  training  to  have  little  or  no  value.  TRUE  pilots  stated  that  both  the  manned  and  computer¬ 
generated  threat  pilots  had  perfect  situation  awareness,  flew  aircraft  that  were  too  fast,  had  360deg  radars 
which  could  not  be  defeated,  and  fired  invincible  missiles.  Many  of  these  problems  were  found  to  originate 
from  aircraft  and  radar  models  used  in  the  ATES  which  were  generated  from  unclassified  and  widely 
available  reference  sources.  While  each  parameter  used  in  these  models  may  have  been  only  slightly 
optimistic,  combining  them  all  into  a  set  of  threat  models  produced  an  unbeatable  foe.  Adjusting  the  threat 
model  parameters  using  better  data  or  pilot  acceptance  greatly  increased  Multirad  training  effectiveness 
ratings  during  the  remainder  of  the  TRUE. 

One  difficulty  which  could  not  be  corrected  was  the  infrared  guided  missile  model  used  by  the 
manned  opposing  forces  pilots  flying  CETs.  This  .model  was  originally  developed  for  intercept  training 
against  a  nonresponding  target.  If  the  GET  pilot  has  correctly  positioned  his  aircraft  with  respect  to  the 
target,  the  missile  scores  a  kill.  When  integrated  into  Multirad,  however,  the  target  aircraft's  pilot  would 
attempt  to  defeat  the  missile  by  effecting  countermeasures.  These  countermeasures  had  no  effect  on  the 
CET's  missile.  Pilots  greatly  objected  to  this  aspect  of  Multirad  simulation  since  it  prevented  them  from 
practicing  their  skills  at  missile  defense.  The  only  corrective  measure  possible  during  the  TRUE 
evaluation  was  to  brief  the  GET  pilots  to  hold  their  shots  until  the  probability  of  a  kill  was  very  high.  This 
solution  satisfied  no  one. 

Air  Weapons  Controller  Station  -  Although  the  SGGENTS  station  provides  only  a  functional 
representation  of  an  AWAGS  or  GGI  display  rather  than  a  high  fidelity  physical  simulation,  the  controllers 
who  participated  in  the  TRUE  gave  uniformly  high  ratings  to  the  training  provided  by  Multirad.  During 
interviews,  controllers  stated  that  the  SGGENTS  station  did  present  a  number  of  simisms  to  which  they 
quickly  adapted.  The  most  significant  simism  was  that  the  simulated  radar  was  too  good.  Aircraft 
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altitudes  were  identified  too  precisely  and  there  were  no  blind  areas  behind  terrain  features.  The  training 
value  in  Multirad  came  from  the  opportunity  to  practice  several  setups  within  a  simulator  period  and  then 
to  debrief  these  engagements  with  the  pilots  while  watching  the  videotapes  and  listening  to  their  radio 
calls. 


Debriefing  Station  -  Pilots  and  controllers  rated  the  computer  system  for  controlling  the 
synchronized  videotapes  as  being  overly  complex.  Only  a  few  participants  actually  learned  to  use  the 
system  to  its  full  advantage.  Redesigning  the  system  to  make  it  fighter  pilot-friendly  is  a  high  priority  item 
for  improving  Multirad.  Aside  from  the  complexity,  however,  pilots  agreed  with  the  controllers  that 
reviewing  videotapes  of  the  engagement  added  greatly  to  the  value  of  Multirad  training.  In  particular, 
seeing  the  overhead  view  together  with  their  own  radar  allowed  pilots  to  evaluate  the  effectiveness  of  their 
actions  in  context  of  the  entire  mission. 

Interest  in  Additional  Training 

Pilots  rated  their  interest  in  receiving  additional  training  on  each  of  30  tasks  using  a  scale  from  1  = 
additional  training  is  not  desirable  to  5  =  additional  training  is  extremely  desirable.  Mean  ratings  are 


presented  in  Figure  14. 

The  30  tasks  on  the  figure  are  coded; 

TACTICAL  FRM 

Tactical  formation 

VISUAL  LOW 

Visual  low  level  flight 

SEPARATION 

Separation  tactics 

VISUAL  ID 

Visual  identification  of  target  aircraft 

LOWALTTAC 

Low  altitude  tactics 

DEBRIEFING 

Mission  debriefing 

EGRESS  TAC 

Egress  tactics 

INTRAFLT  COM 

Intrafiight  communication 

TWO-SHIP  TAC 

Two-ship  tactics 

COM  JAMMING 

Communication  jamming 

TEWS 

Tactical  Electronic  Warfare  System  assessment 

INTERCEPT 

Tactical  intercept 

MISSILE 

Missile  employment 

ECM 

Employ  ECM/ECCM 

Al  RESPONSE 

Reaction  to  airborne  interceptors 

MUTUAL  SUPT 

Mutual  support 

WEATHER 

All-weather  employment 

CHAFF 

Employ  chaff/flares 

VISUAL  LOOK 

Visual  lookout 

EID 

Electronic  identification  of  target  aircraft 

TACTICS/PLAN 

Tactics/mission  planning  and  briefing 

ESCORT  TACTICS 

Escort  tactics 

RADAR  LOOK 

Radar  lookout 

WORK  W/GCI 

Work  with  AWACS/GCI 

SAM  DEFENSE 

Reaction  to  surface-to-air  missiles  (SAMs) 

RADAR  SORT 

Radar  employment/sorting 

BVR 

Beyond-visual-range  employment 

ALL-ASPECT  D 

Alt-aspect  defense 

DACT 

Dissimilar  air  combat  training 

FOUR+  BOGEYS 

Multibogey,  four  or  more 

The  five  tasks  with  the  lowest  interest  ratings  are  primarily  visual  tasks.  The  tasks  with  the  highest  rated 
interest  in  receiving  additional  training  are  tasks  which  can  usually  be  practiced  only  in  large  exercises  or 
cannot  be  practiced  except  in  simulation,  e.g.  defense  against  SAMs.  This  result  is  in  agreement  with 
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Houck  et  al.  (1991)  who  found  that  pilots  were  most  interested  in  receiving  additional  training  for  tasks 
which  are  least  frequently  practiced  in  the  aircraft.  Interest  is  low  for  tasks  such  as  tactical  formation  or 
separation  tactics  which  are  practiced  on  all  air-to-air  sorties. 


I- 


Not 

Desirable 


Pilot  rating 


Extremely 

Desirable 


Figure  14.  Pilot  Ratings  of  Interest  in  Receiving  Additional  Training  for  30  Mission  Activities. 

Value  of  Multirad  and  Current  Unit  Training 

Using  the  scale  of  1  =  unacceptable  to  5  =  excellent,  pilots  rated  the  value  of  their  current  unit 
training  and  Multirad  training  for  each  of  the  30  tasks.  The  lowest  and  highest  rated  tasks  for  unit  and 
Multirad  training  are  listed  in  Table  6.  While  some  tasks,  notably  radar  lookout,  were  rated  highly  for  both 
Multirad  and  current  unit  training,  more  tasks  were  given  high  ratings  for  one  training  environment  and  low 
ratings  for  the  other.  For  example,  visual  lookout  and  tactical  formation  were  the  lowest  rated  tasks  for 
Multirad  but  among  the  highest  rated  tasks  for  current  unit  training.  Likewise,  work  with  GCI  controllers 
and  engagements  against  four  or  more  bogeys  were  among  the  lowest  rated  tasks  for  current  unit  training 
and  among  the  highest  rated  tasks  for  Multirad. 

The  differences  between  the  mean  ratings  for  unit  and  Multirad  training  are  plotted  on  Figure  15. 
Tasks  which  were  rated  much  higher  in  the  current  unit  than  in  Multirad  are  primarily  visual.  These  tasks 
were  also  among  the  lowest  rated  tasks  for  interest  in  receiving  additional  training.  Tasks  which  were 
rated  higher  in  Multirad  than  in  the  current  unit  are  tasks  which  are  not  frequently  practiced  outside  of 
large-scale  exercises.  Also,  tasks  rated  higher  for  Multirad  training  were  among  the  highest  rated  tasks 
for  interest  in  receiving  additional  training.  This  finding  is  also  in  agreement  with  Houck  et  al.  (1991)  who 
reported  that  pilots  rated  multiplayer,  simulator-based  training  higher  than  unit  training  for  tasks  which 
cannot  be  practiced  in  aircraft  due  to  safety,  cost,  and  security  restrictions. 
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Task 


Table  6.  Lowest  and  Highest  Rated  Tasks  for  Current  Unit  and  Multirad  Training. 


Current  Unit 

Multirad 

Lowest  rated 

SAM  defense 

Visual  ID 

tasks 

Com  jamming 

Tactical  formation 

ECM/ECCM 

Com  jamming 

Work  w/GCI 

Visual  lookout 

Four+  bogeys 

Visual  low  altitude 

Highest  rated 

Visual  lookout 

Radar  lookout 

tasks 

Tactics/planning 

Work  w/GCI 

Mutual  support 

Radar  sorting 

Tactical  formation 

BVR  employment 

Radar  lookout 

Four+  bogeys 

(UPPERCASE  task  ^ 
rndicatQs  hign  interest 
in  additional  training) 

tactical  frm 
visual  look 
visual  id 
mutual  supt 
visual  low 
two-ship  tac 
ALL'ASPECT  D 
low  alt  tac 
TACTICS/PLAN 
separation 
□ACT 
RADAR  LOOK 
ESCORT  TAC 
debriefing 
ai  response 
egress  tac 
intercept 
com  jamming 
eid 

intraflt  com 
weather 
missile 
RADAR  SORT 
BVR 
tews 
chaff 
ecm 

SAM  DEFENSE 
WOEKW/GCI 
FOUR+ BOGEYS  ^ 


-3-2-10  12  3 

Current  Unit  Multirad 

Difference  Between  Multirad  and  Unit  Training 

Figure  15.  Differences  Between  Pilot  Ratings  of  Training  Effectiveness  for  Multireel  and  Current  Unit 

Training  for  30  Mission  Activities. 
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Similar  ratings  of  current  unit  and  Multirad  training  were  collected  from  air  weapons  controllers  for 
the  subset  of  ten  tasks  which  were  applicable.  The  differences  between  the  mean  ratings  for  unit  and 
Multirad  training  are  plotted  on  Figure  16.  For  all.  tasks,  Multirad  training  was  rated  higher  than  current  unit 
training  with  highest  ratings  for  those  rarely  practiced;  i.  e.,  dissimiliar  air  combat  tactics,  debriefing  with 
the  pilots,  and  engagements  against  multiple  bogeys. 


TACTICS/PLAN 
RADAR  SORT 
INTRAFLT  COM 
INTERCEPT 
^  TWO-SHIP  TAC 
BVR 

ESCORT  TAC 
DACT 
DEBRIEF 
FOUR+  BOGEYS 


-3-2-10  1  2  3 

Current  Unit  Multirad 

Difference  Between  Multirad  and  Unit  Training 

Figure  1 6.  Differences  Between  Air  Weapons  Controller  Ratings  of  T raining  Effectiveness  for  Multirad 

and  Current  Unit  training  for  10  Mission  Activities. 

Mission  Performance 

During  the  TRUE,  each  team  typically  flew  three  OCA  simulator  sessions  and  three  DCA 
simulator  sessions  with  three  or  four  setups  per  session.  Mission  performance  for  each  OCA  and  DCA 
session  was  summarized  as  average  number  of  enemy  aircraft  shot  down  (kills)  per  setup  and  average 
number  of  F-15s  shot  down  (morts)  per  setup.  These  data  were  compared  for  less  experienced  pilots 
(wingman  or  two-ship  flight  lead,  787  median  flight  hours)  and  more  experienced  pilots  (four-ship  flight 
leads  and  instructor  pilots,  1975  median  flight  hours).  The  data  are  summarized  on  Figure  17  (DCA)  and 
Figure  1 8  (OCA). 


DISCUSSION 
Lessons  Learned 

The  objective  of  the  TRUE  was  to  determine  whether  the  benefits  observed  in  the  training  utility 
studies  at  MCAIR  could  be  replicated  with  a  lower  cost,  distributed  interactive  simulation.  Overall,  the 
TRUE  demonstrated  that  multiship  air  combat  simulation  can  be  successfully  conducted  using  a  SIMNET- 
compatible  approach.  The  lessons  learned  from  TRUE  provide  guidance  for  the  development  of  future 
systems  and  further  modifications  to  the  Multirad  system. 
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Figure  17.  Mission  Performance  for  the  TRUE  DCA  Scenarios:  (a)  average  enemy  kills  per  session,  and 
(b)  average  F-15  losses  (morts)  per  session  for  less  and  more  experienced  pilots. 
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Figure  18.  Mission  Performance  for  the  TRUE  OCA  Scenarios:  (a)  average  enemy  kills  per  session,  and 
(b)  average  F-15  losses  (morts)  per  session  for  less  and  more  experienced  pilots. 

Functional  vs  Physical  Fidelity 

Boyle  and  Edwards  (1992)  point  out  that,  "A  real  program  killer  is  unmet  pilot  expectations.  If  you 
are  not  simulating  it,  don't  try  to  make  it  look  like  you  are,"  (p.  496).  This  lesson  was  repeated  during  the 
TRUE  in  that  functions  which  were  incompletely  modeled  but  were  central  to  the  missions  being  practiced 
raised  the  most  serious  objections.  Notably,  the  lack  of  currency  between  the  pilot's  aircraft  and  the 
simulated  F-15  software  raised  howls  of  protest.  Pilots  were  compelled  by  the  simulation  to  practice  using 
tactics  that  they  would  not  use  if  they  were  to  go  to  war  tomorrow.  In  this  case,  reduced  fidelity  was  not 
good  enough.  Compare  this  response  to  the  glass  cockpit  MDRC.  The  front  panel  was  a  CRT,  there 
were  no  rudder  pedals,  and  most  switches  were  missing.  None  of  these  faults  affected  the  mission  and 
they  caused  no  objections.  Systems  which  were  critical  to  the  mission,  hands-on-throttle-and-stick 
(HOTAS)  controls  and  the  displays  used  in  air  combat,  were  high  fidelity  simulations.  The  net  effect  was  a 
fully  acceptable  trainer,  except  for  the  software.  Air  weapons  controllers  using  the  SCCENTS  station  had 
a  similar  experience.  The  physical  similarity  between  the  SCCENTS  and  an  actual  AWACS  or  GCI  station 
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was  limited  to  the  mission  critical  information  on  the  display.  Controllers  adapted  to  the  simisms  caused 
by  the  noncritical  elements  of  the  SCCENTS  and  rated  the  system  as  providing  high  value  training. 

integration  of  existing  systems 

The  CET  used  as  a  manned  opponent  in  the  TRUE  is  a  part  task  trainer  designed  to  teach  the 
shooter  to  intercept  an  air  target  and  to  put  his  aircraft  into  an  optimal  firing  position.  The  objective  of  this 
part-task  trainer  is  to  teach  the  pilot  to  make  the  intercept  and  to  take  a  good  shot.  The  missile  model 
provides  feedback  to  the  pilot  about  his  intercept:  a  good  shot  gets  a  kill  and  a  bad  shot  misses.  This 
missile  model  completely  fulfills  the  CET's  training  objectives.  The  objectives  of  Multirad  training, 
however,  include  training  the  F-15  pilot  to  defend  against  air-to-air  missiles.  The  infrared  (IR)  missile 
models  used  by  the  ATES  were  sensitive  to  flares  and  the  target  aircraft's  throttle  setting  so  that  the  F-15 
pilot  could,  if  he  was  quick  enough,  defeat  an  ATES  missile.  CET  missiles  were  not  designed  to  provide 
defensive  training  for  the  target  aircraft's  pilot  and  were  therefore  not  responsive  to  flares  or  other 
countermeasures.  The  CET's  missile  is  not  a  low  fidelity  model  within  its  intended  application.  Integrated 
into  Multirad,  however,  the  CET  missile  was  unacceptable.  Integrating  existing  systems  requires  very 
careful  consideration  of  each  system’s  original  objectives  and  how  that  system  operates.  Networking 
existing  systems  will  support  effective  training  only  if  the  objectives  of  the  integrated  system  are  clearly 
stated  and  the  capabilities  of  the  individual  components  are  evaluated  in  terms  of  these  objectives. 

Multirad  Network  Limitations 

The  simulation  network  was  not  a  limiting  factor  on  the  current  Multirad  system.  Network 
utilization  for  the  TRUE  effort  never  exceeded  3.4%  of  the  full  capacity  of  ethernet.  With  bandwidths  of 
100  Megabits/s  available  for  FDDI,  the  network  should  continue  to  fulfill  the  requirements  of  mission 
rehearsal  and  team  training.  The  limiting  factors  found  during  the  TRUE  were:  host  to  NIU  interface,  NIU 
processing  capability,  and  host  computer  processing  capability.  The  MDRC  ICDs  were  developed  to 
maximize  the  amount  of  valuable  information  passed  to  and  from  the  network.  As  the  number  of  entities 
on  the  network  increases,  the  simulator  and  NIU  spend  more  time  transferring  and  processing  this  data. 
Data  minimization  methods  must  be  developed  to  optimize  not  only  the  data  that  is  sent  to  and  from  the 
host  but  also  the  frequency  of  these  transfers.  Also,  the  network  analysis  showed  the  NIU  spent  the 
largest  amount  of  time  building  up  the  host  data  buffer.  For  example,  it  took  an  average  49.5%  (24.75  ms) 
of  the  total  frame  for  the  NIU  to  build  the  MDRC’s  data  buffers.  The  primary  restriction  encountered  in 
setting  up  the  TRUE  scenarios,  however,  was  the  physical  limitations  of  the  host  simulators'  computing 
power. 

Test  Procedures  and  Configuration  Control 

As  the  integration  process  progressed,  it  became  apparent  that  more  testing  and  configuration 
control  were  needed.  The  time  between  the  TRUE  scenarios  was  not  always  adequate  to  thoroughly  test 
changes  to  the  overall  system.  Additionally,  the  integration  effort  was  accomplished  by  four  separate 
contractors  and  the  government.  Some  problems  occurred  due  to  miscommunications  between  the 
integration  teams  while  others  were  a  result  of  misinterpretations  of  the  protocols  and  ICDs,  Still  others 
were  due  to  poor  configuration  control  of  software.  Problems  encountered  and  fixed  would  sometimes 
reappear  in  scenarios.  Thorough  test  procedures  and  rigorous  configuration  control  are  crucial  for  an 
integration  effort  of  this  type. 

Visual  Displays 

A  major  limitation  on  the  value  of  Multirad  training  resulted  from  the  visual  displays.  While  wide 
field-of-view  displays  were  rated  by  pilots  as  necessary  for  effective  training,  inadequacies  in  the  visual 
display  systems  reduced  pilot  situation  awareness  and  induced  simulator-unique  behaviors.  Pilots  could 
not  maintain  tactical  formation  or  count  on  mutual  visual  support.  Although  the  DART  had  poorer 
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resolution  than  the  dome,  pilots  preferred  the  DART  due  to  its  greater  contrast  and  larger  instantaneous 
field  of  view.  While  their  mission  performance  was  often  successful,  pilots  asserted  that  their  tactics  were 
sometimes  different  from  aircraft  tactics  leading  to  opportunities  for  negative  transfer  of  training. 
Representative  comments  from  TRUE  pilots  include,  "We  need  two  DARTs  so  we  can  keep  sight  of  each 
other.  With  current  systems  we  are  developing  simisms,"  or,  "He  who  sees  will  have  better  SA  [situation 
awareness],"  and,  the  most  telling  comment  was,  "I  would  have  died  less  if  I  could  have  seen  more." 

Multiship  Training 

TRUE  was  conducted  as  an  engineering  test  of  the  Multirad  system.  However,  TRUE  also 
provided  the  opportunity  to  gather  behavioral  data  on  training  interests  and  the  potential  applications  of 
mutiship,  simulator-based  training  for  air  combat.  Overall,  the  results  of  TRUE  are  in  agreement  with  the 
findings  of  Houck  et  al.  (1991).  Although  the  engineering  development  simulator  system  used  by  Houck  et 
al.  at  MCAIR  had  more  capability  than  the  present  Multirad  system,  TRUE  pilots  rated  simulator  training 
higher  than  current  unit  training  for  a  similar  list  of  tasks.  These  include  tasks  which  can  be  performed 
single-ship  but  are  infrequently  practiced  such  as  radar  sorting  against  multiple  targets.  The  high  rated 
tasks  also  include  skills  which  can  only  be  practiced  within  the  context  of  a  team.  These  tasks  include 
work  with  a  GCI  controller  and  operations  against  four  or  more  bogeys. 

Dion  and  Bardeen  (1990)  predict  that  the  major  benefit  of  multiplayer  combat  training  will  be  the 
development  of  team  skills.  These  benefits  will  come  from,  "training  situations  in  which  the  team  learns  to 
develop  adaptive  teamwork  skills  required  in  real  time  for  uncertain  problems,  and  an  expanded  repertoire 
of  team  experiences,"  (p.  468).  It  is  unclear,  however,  whether  the  benefits  of  multiplayer  simulation  come 
from  development  of  team  skills  or  from  the  development  of  individual  skills  at  tasks  which  can  only  be 
practiced  within  a  team  context.  Crane  (1992)  predicts  that  training  will  be  of  most  benefit  for  skills  which 
pilots  have  had  the  least  opportunity  to  practice.  This  prediction  is  based  on  cognitive  models  of  expertise 
which  assert  that  a  journeyman  lacks  the  expert's  extensive  knowledge  base  and  is  unable  to  quickly 
recognize  the  significant  elements  within  a  mission  and  to  select  an  appropriate  response.  The  idea  that 
practice  will  be  of  most  benefit  for  the  least  practiced  skills  is  supported  by  two  sources  of  evidence  within 
TRUE. 


Figure  19.  Mean  Effectiveness  Ratings  Assigned  to  Multirad  Training  Compared  to  Total  Flight  Hours. 
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Pilot  ratings  of  Multirad  training  value  were  averaged  over  the  30  tasks  listed  on  Figure  15  and 
compared  to  overall  experience.  There  is  a  significant,  negative  correlation  between  flight  hours  and 
average  rating  of  Multirad  training  effectiveness  (r  =  -.62),  (see  Fig.  19).  Overall,  the  ratings  of  the  more 
experienced  pilots  were  lower  and  variable.  Less  experienced  pilots  gave  consistently  high  ratings  for 
Multirad  training  and  gave  significantly  higher  ratings  than  the  more  experienced  pilots  for  tactics,  mission 
planning,  and  mutual  support.  High  experience  pilots  gave  Multirad  higher  ratings  than  the  less 
experienced  pilots  only  for  chaff  employment.  During  interviews,  wing  pilots  and  less  experienced  two- 
ship  flight  leads  stated  that  if  given  the  opportunity,  they  would  use  the  Multirad  system  "everyday."  On 
the  other  hand,  instructor  pilots  and  four-ship  flight  leaders  stated  that  they  found  the  Multirad  system 
interesting  but  that  the  training  was  not  particularly  valuable.  However,  these  pilots  also  stated  that  they 
would  be  very  interested  in  using  a  system  with  4  F-15s  and  8-12  threat  aircraft. 

Pilot  performance  data  from  the  TRUE,  although  limited,  also  tends  to  support  the  idea  that 
Multirad  training  is  most  valuable  for  infrequently  practiced  tasks.  In  the  TRUE  training  scenarios,  the 
DCA  mission  was  less  complex  and  more  familiar  than  the  OCA  scenarios.  As  can  be  seen  in  Figure  17, 
the  more  experienced  pilots  showed  relatively  high  levels  of  performance,  both  kills  and  morts,  and  did  not 
improve  over  the  three  opportunities  to  practice  this  mission.  The  less  experienced  pilots,  however, 
initially  performed  more  poorly  than  the  more  experienced  pilots  but  improved  over  the  week  in  both  kills 
and  morts.  For  the  more  complex  and  less  familiar  OCA  scenarios,  the  more  experienced  pilots  showed 
improvements  in  kills  (Figure  18).  The  less  experienced  pilots  did  not  improve  their  kill  performance  and 
neither  group  improved  on  morts.  This  suggests  that  the  mission  scenario  was  overly  complex  and  that  a 
building  block  approach  should  have  been  employed.  Since  the  mission  activities  in  TRUE  which  were 
highly  rated  for  Multirad  training  are  both  team  tasks  and  infrequently  practiced,  no  conclusion  can  be 
drawn  regarding  Dion  and  Bardeen's  (1990)  prediction  that  the  benefit  of  multiplayer  simulation  will  be 
development  of  team  skills. 


Conclusions 

There  is  a  growing  realization  that  combat  mission  training  cannot  be  fully  accomplished  using  in¬ 
flight  training.  The  evaluations  conducted  at  MCAIR  and  AL/HRA  demonstrated  the  experienced  pilots 
and  air  weapons  controllers  perceive  multiship  air  combat  simulations  as  providing  additional  air  combat 
training  opportunities  beyond  current  unit  training.  Multiship  simulation  provides  pilots  and  controllers  the 
means  to  practice  tasks  which  cannot  be  practiced  in  the  aircraft  due  to  cost  and  peacetime  training 
restrictions. 

Multiship  air  combat  simulations  using  a  super-minicomputer,  shared  memory  architecture  were 
successfully  transitioned  to  a  distributed  microprocessor  architecture  using  a  communications  system 
which  is  compatible  with  other  military  trainers.  While  this  transition  has  provided  high-level  training  using 
low-cost  devices,  several  limitations  remain.  Most  notably,  the  out-the-window  visual  simulation  cannot 
provide  the  level  of  resolution  necessary  to  judge  aspect  and  closure  of  air  targets  at  realistic  ranges. 
Also,  the  opportunity  to  integrate  existing  training  devices  into  a  multiplayer  network  may  create  new 
difficulties.  Each  device's  capabilities  and  level  of  fidelity  needs  to  be  carefully  considered  with  respect  to 
the  training  objectives  of  the  integrated  system.  Ill-considered  choices  may  degrade  rather  than  enhance 
the  quality  of  training. 
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